Skip to content

feat(trtllm): select conversation affinity rank owner - #12225

Merged
krishung5 merged 7 commits into
mainfrom
krish/trtllm-conversation-explicit-rank
Jul 30, 2026
Merged

krishung5 merged 7 commits into
mainfrom
krish/trtllm-conversation-explicit-rank

Conversation

@krishung5

@krishung5 krishung5 commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

Add an opt-in path for Dynamo to own the initial attention-DP placement of a conversation while TensorRT-LLM preserves the resulting conversation affinity.

This is the Dynamo-side companion to NVIDIA/TensorRT-LLM#16815 and follows up on the design discussion in #11609.

Details:

  • Add --conversation-affinity-dp-rank-source {engine,dynamo} and the equivalent DYN_ENGINE_CONV_AFFINITY_DP_RANK_SOURCE environment variable.
  • Keep engine as the default, preserving the existing behavior where TensorRT-LLM load-balances the first turn.
  • In dynamo mode, forward both ConversationParams and Dynamo's strict attention_dp_rank when a stable session ID and router-selected rank are available.
  • Preserve TensorRT-LLM's no-ID balancing fallback when the request has no session ID.
  • Add CLI/environment/default and request-handler coverage for both rank owners.

dynamo mode requires NVIDIA/TensorRT-LLM#16815 (or an equivalent TRT-LLM change) so the explicit first-turn rank is recorded as the conversation binding and later conflicting hints do not move the conversation.

Where should the reviewer start?

  • components/src/dynamo/trtllm/request_handlers/handler_base.py
  • components/src/dynamo/trtllm/backend_args.py
  • components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py

Validation:

  • pre-commit run --files ...
  • python3 -m compileall -q ...
  • PYTHONPATH=components/src python3 -m pytest -q components/src/dynamo/trtllm/tests/test_trtllm_unit.py components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py components/src/dynamo/trtllm/tests/test_conversation_affinity.py (6 passed, 3 skipped; CUDA/TRT-LLM-dependent modules are skipped on the local CPU-only host and will run in CI)
  • git diff --check

Related Issues

🚫 This PR is NOT linked to an issue:

  • Confirmed — no related issue

Summary by CodeRabbit

  • New Features

    • Added configuration to choose whether TensorRT-LLM or Dynamo controls initial attention-DP placement when conversation affinity is enabled.
    • Added CLI and environment-variable support for selecting the placement source.
    • Dynamo-controlled placement now forwards the selected rank for requests with a conversation identifier; existing engine-controlled behavior remains the default.
  • Tests

    • Added coverage for CLI, environment-variable, default configuration, and conversation-affinity routing behavior.

@datadog-official

datadog-official Bot commented Jul 27, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 28.16% (-14.46%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 89b5458 | Docs | Datadog PR Page | Give us feedback!

Signed-off-by: krishung5 <krish@nvidia.com>
@krishung5
krishung5 force-pushed the krish/trtllm-conversation-explicit-rank branch from d9b78fd to 41b9f4e Compare July 28, 2026 16:21
@krishung5
krishung5 marked this pull request as ready for review July 28, 2026 16:25
@krishung5
krishung5 requested review from a team as code owners July 28, 2026 16:25

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

Conversation affinity routing

Layer / File(s) Summary
Configuration and handler wiring
components/src/dynamo/trtllm/backend_args.py, components/src/dynamo/trtllm/request_handlers/handler_base.py, components/src/dynamo/trtllm/workers/llm_worker.py
Adds the engine/dynamo rank-source option, stores it in configuration, and passes it into request handling.
Initial placement selection
components/src/dynamo/trtllm/conversation_affinity.py, components/src/dynamo/trtllm/request_handlers/handler_base.py
For Dynamo-owned placement, forwards the router’s DP rank with strict scheduling parameters when a conversation session ID exists; engine-owned placement remains the default.
Routing and configuration tests
components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py, components/src/dynamo/trtllm/tests/test_trtllm_unit.py, components/src/dynamo/trtllm/tests/utils.py
Tests CLI, environment, and default configuration handling, Dynamo rank forwarding, no-session behavior, and updated mock configuration setup.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description covers overview, details, review start, and validation, but the required Related Issues section is inconsistent and incomplete. Pick one Related Issues path, remove the other, and either add a Closes/Relates to #XXXX link or leave only the confirmed-no-issue checkbox.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: letting Dynamo select the conversation-affinity rank owner in TRT-LLM.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/trtllm/request_handlers/handler_base.py`:
- Around line 1156-1172: Add a runtime capability check around the
dynamo-specific branch in the request-handler flow, rather than relying only on
CONVERSATION_PARAMS_AVAILABLE. When conversation-affinity dp-rank sourcing is
configured as “dynamo,” require the engine capability corresponding to
persistent first-turn conversation binding from TensorRT-LLM#16815; otherwise
reject the configuration or fall back to “engine” before applying
SchedulingParams in the shown branch.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 18c0358d-1246-44ce-8db7-9314632bf212

📥 Commits

Reviewing files that changed from the base of the PR and between 16e1cf1 and 41b9f4e.

📒 Files selected for processing (7)
  • components/src/dynamo/trtllm/backend_args.py
  • components/src/dynamo/trtllm/conversation_affinity.py
  • components/src/dynamo/trtllm/request_handlers/handler_base.py
  • components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py
  • components/src/dynamo/trtllm/tests/test_trtllm_unit.py
  • components/src/dynamo/trtllm/tests/utils.py
  • components/src/dynamo/trtllm/workers/llm_worker.py

Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
@ai-dynamo ai-dynamo deleted a comment from coderabbitai Bot Jul 28, 2026
@krishung5

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@krishung5
krishung5 requested a review from a team as a code owner July 28, 2026 17:24
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 28, 2026
@github-actions

github-actions Bot commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor

@krishung5
krishung5 enabled auto-merge (squash) July 29, 2026 18:34
@krishung5
krishung5 merged commit a387b06 into main Jul 30, 2026
103 checks passed
@krishung5
krishung5 deleted the krish/trtllm-conversation-explicit-rank branch July 30, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::trtllm Relates to the trtllm backend documentation Improvements or additions to documentation feat size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants